Repository navigation
feat: Add experimental support for flag overrides - #580
Merged
Merged
Conversation
<!-- TERMAG_PR_BODY_BEGIN -->
## Summary
Adds `ldclient.impl.integrations.files.filedata`, the file reading,
parsing, and merging logic that the file-based override source described
by the OVERRIDE specification is built on. The specification requires
ordered multi-file merging with duplicate keys handling, a reload that
keeps the last good data across a malformed edit and retries, debouncing
of change notifications, a polling change detector as well as a watching
one, and correct handling of a configured file that does not exist yet.
This module provides those pieces; the override source itself is added
in a later PR of the stack.
**The existing file data sources are not changed.** The FDv1 file update
processor (`Files.new_data_source`) and the FDv2 file initializer and
synchronizer (`Files.new_data_source_v2`, `datasystem.file_ds_builder`)
keep their current implementation and behavior in every respect: the
`flagValues` expansion (an on flag with fallthrough variation 0 and
version 1, evaluated with a `FALLTHROUGH` reason), the version fallback
that stamps version 1 on entries without one, the failure messages and
their levels, the FDv2 synchronizer's `OFF` state on a failed initial
load and its `INVALID_DATA` error kind, and the polling and watching
rules. A new test module, `test_file_data_sources_pinned_behavior.py`,
pins each of those behaviors so the override feature stays purely
additive. The baseline test files for the existing sources are
unmodified.
What the new module provides for the override source:
- Document parsing. The file name chooses the parser: a name that ends
in `.yaml` or `.yml`, in any case (including a file named just `.yaml`),
is YAML, which needs `pyyaml`, and any other file is JSON, parsed by the
standard JSON parser even when `pyyaml` is installed. The content is
never inspected to choose a parser. Files must be UTF-8; a leading byte
order mark is ignored, and UTF-16, UTF-32 or invalid UTF-8 fails to
parse. The format decides whether an empty file is valid: an empty or
white-space JSON file is not a document and fails to parse (`the file
has no content, which is not a JSON document`), as does JSON that is not
an object, such as `null`; YAML that is empty, only comments, `---`, or
`null` has no entries. A parse error names the format the name chose.
Flag and segment definitions are decoded into the model classes while
the file is read, so an invalid definition fails that load. The map key
is the key of a flag or segment definition for every purpose: the
definition's own key is set to it, and duplicates are detected by it, so
evaluations report the flag by that key. A definition may omit its
version, which is filled in as 1.
- Merging. Files are combined in the configured order. The duplicate
keys handling is `fail` (the load fails) or `ignore` (the first file's
entry is kept). The result records how many entries each file supplied.
- `Reloader`. Serializes reloads, debounces change signals with a settle
window that each signal extends, keeps the last good data by not
applying a failed load, retries a failed load after a bounded delay so a
file observed mid-write recovers without another notification, reports
an identical failure once, skips an application whose files are
byte-identical to the last applied ones, and applies a success after a
failure even when the contents did not change. A configured file that
does not exist can contribute no data instead of failing the load. A
load reads, parses and classifies every configured file, even after one
fails, then fails with the first failure (logging the others at Debug
level), then holds files back (below), then merges. The skip-unchanged
digest covers each path, whether its file exists, and its bytes, since
the same bytes in a file of another name can be in another format.
Closing does not wait for an in-flight reload. An exception raised by
`apply` or `on_error` is reported like any other load failure and
retried, and nothing from that load is remembered.
- In-place rewrites. Applications and editors rewrite files in place:
they truncate the file, then write the new content, and a load in
between finds it empty. A file whose content is in effect and that a
load finds empty, white space only, or parsed to nothing (YAML with only
a comment or `---`) is held back: the load keeps the current data, logs
at Debug level and reports nothing, and the other files' changes wait
too, since a load applies all files or none. `{}` is a document and is
applied at once. The hold works as follows:
- The file is held until it has stayed so, unwritten, for the retry
delay. Its state (modification time and size, read through symbolic
links) is compared only with itself, never with the clock, and a change
starts the delay over.
- The clock is read after the file's own read, so a slow read of another
file does not change its delay.
- The hold asks for a retry when the first held file is due, plus a
tenth of the retry delay; the write's change signal starts a load
sooner.
- Each file is held for at most five retry delays after its first hold,
so a file kept written while empty, or files emptied in turn, are
applied in the end. A file's bound starts over only at a load that holds
nothing back, whatever that load's outcome.
- Content, even content that cannot be parsed, or a missing file ends a
file's empty state; a failed read keeps it.
- A file whose content is not in effect (a placeholder that was always
empty, a file whose deletion was applied and that is then created empty,
a file whose emptiness was applied) is never held, and with retries off
nothing is held. A file that is deleted and created empty again before a
load applies the deletion, in the same debounce window or while another
file fails or is held, still has its content in effect, so it is held
like a file rewritten in place.
- After the delay, empty YAML is applied with no entries, and empty JSON
fails to parse and keeps the data.
- `Poller`. Compares modification time and size on an interval, so a
same-size rewrite and a same-time size change are both detected, as are
files that appear or disappear.
- `Watcher`. Uses the `watchdog` package. It watches the directory of
each file so an absent file is picked up when it appears, matches the
destination of a move event so a file written by rename is detected,
retries a directory that does not exist yet, and reacts only to
notifications that can change a file's content or presence (the watchdog
inotify mask also reports a file being opened or read). A failed
observer start is retried every second, and a watched directory that is
deleted is watched again once it exists, with one change signaled. A
configured file that is a symbolic link is followed: the directory of
the link's target is watched for the target's path, and the links are
resolved again after every reported change, after a retry sets up
watches, and when a directory that holds a target is deleted, as a
Kubernetes ConfigMap update does. Watching does not follow a link
further along the chain that is pointed elsewhere while its old target
stays, a configured directory that is itself a link and is pointed
elsewhere, or a configured directory that is renamed and replaced; the
documentation points such deployments, like a `current` link to a
release directory, to polling, which is the default. A link that cannot
be resolved, for example because its target's directory does not exist
yet or was deleted, is resolved again on the retry interval until its
target is watched, and one change is then signaled. A relative link is
joined to the real path of the directory that holds it, so a `..` in it
is right when a directory on the way is itself a link.
Not changed: a read that sees a YAML file cut off part way can still
parse, with fewer entries or a different value, and is applied until the
next reload; the override source's docs recommend replacing a YAML file
in one operation.
Tests cover parsing (format by name, empty and non-object JSON, empty
and null YAML, byte order marks and other encodings), merging, loading,
the reloader (initial load, failure retention, debounce coalescing and
window extension, retry with and without further signals,
identical-failure and first-failure reporting, skip-unchanged and
recovery, serialization of concurrent reloads, close semantics, worker
thread lifecycle), the hold (on an injected clock, with a one-hour retry
delay so no real retry fires: the exact delay, restarts on a state
change and through a symbolic link, quick successive rewrites, files
emptied together and in turn, the per-file bound and when it starts
over, failures before and after a held file, slow reads, the retry each
hold asks for; and on the real clock, that the retries fire), the
poller, and the watcher (including symbolic links to files in the same
and in another directory, a link that is replaced, a ConfigMap-style
update, a relative link in a directory reached through another link, and
a target directory that appears late or is recreated), plus the pinned
behavior of the existing sources. The reloader tests rewrite their files
in place with a 30 ms pause, through `ldclient/testing/file_util.py`.
The in-place rewrite handling was reviewed in eight rounds as #560 (now
closed, its changes moved here and into #529): multi-agent and
adversarial reviews of each round, including a randomized end-to-end
stress of about 384 rewrites with no empty snapshot. The final mutation
check ran 59 mutants against `test_filedata.py` and
`test_file_override_source.py`: 54 fail a test, and the 5 survivors are
equivalent in this code (listed in #560). Both test files ran 10 times
in a row with no failure.
Python CI runs for this PR. The PRs above it in the stack get no CI;
they were checked locally.
[SDK-3250](https://launchdarkly.atlassian.net/browse/SDK-3250)
[SDK-3250]:
https://launchdarkly.atlassian.net/browse/SDK-3250?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
<!-- TERMAG_PR_BODY_END -->
<!-- termag: box=i-0575d1c54853058d7 project=Overrides -->
<sub>Opened from termag</sub>
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Overview**
> Introduces **`ldclient.impl.integrations.files.filedata`**, shared
logic for reading JSON/YAML flag/segment files, merging them in order
(`fail` vs `ignore` on duplicate keys), and keeping data fresh via
**`Reloader`** (debounced reloads, failure retention, retries,
skip-unchanged digests, and **hold-back** when an in-effect file is seen
empty mid in-place rewrite), plus **`Poller`** and **`Watcher`**
(watchdog, symlinks, ConfigMap-style dirs) change detectors.
>
> Adds **`ldclient/testing/file_util.py`** for simulated in-place
rewrites and large test suites: **`test_filedata.py`** for the new
module and **`test_file_data_sources_pinned_behavior.py`** to lock
FDv1/FDv2 behavior without changing those implementations in this diff.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4c11d73. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## Summary This is the second step of the flag overrides port described by the OVERRIDE specification. It is based on the shared file data branch because the phases are stacked; retarget to `feat/overrides` once that branch merges. The `FeatureFlag` and `Segment` models gain an override marker, `is_override`. The SDK represents stored entities with typed model classes distinct from their JSON form, so the marker is carried on the type and never in the JSON: `to_json_dict()`, the data kind encoder, and `ModelEncoder` produce the same output for a marked and an unmarked entity, and a persistent store never sees it. `with_override_marker()` returns a marked shallow copy that shares the definition and leaves the source unmarked, so a source can retain and resupply its entities. Only the SDK components that manage override entries (added in the next step) set the marker. The evaluator marks an evaluation as override-affected when any definition it reads carries the marker: the evaluated flag, a prerequisite flag at any depth, or a segment consulted during clause matching. A read counts whether or not the segment matches, including through a negated clause and through a segment referenced by another segment's rule. A definition that cannot be resolved contributes nothing, because nothing was read. The marking propagates upward only: a prerequisite is an evaluation in its own right, its record reflects only the definitions its own subtree read, so an unaffected prerequisite inside a marked evaluation stays unmarked and a marked prerequisite does not mark a sibling. An evaluation that fails with an error reason, including a prerequisite cycle, is still marked when it read an override definition. The marking is reported in two places that always agree: the `overrideAffected` indicator on the evaluation reason, present only when true so an unaffected reason serializes exactly as before, and `EvalResult.override_affected`, which the client and event processing read in the following steps. The `AsyncEvaluator` receives the same changes. Tests cover each marking condition for both evaluators (flag off, fallthrough, and rule match; plain evaluations; malformed flag and prerequisite cycle errors; prerequisite records at depth one and two; unaffected sibling and downward non-propagation; segment reads with and without a match, negated, nested, and through a prerequisite; missing definitions; scalar and indicator agreement) and the model marker (default, marked copy, never serialized, shared definition). [SDK-3250](https://launchdarkly.atlassian.net/browse/SDK-3250) [SDK-3250]: https://launchdarkly.atlassian.net/browse/SDK-3250?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Adds experimental **flag override** tracking: `FeatureFlag` and `Segment` now expose `is_override` and `with_override_marker()` on in-memory models only—the marker never appears in JSON or persisted data. > > Sync and async evaluators set **`EvalResult.override_affected`** and, when true, add **`overrideAffected`** to the evaluation reason (documented on `EvaluationDetail.reason`). An evaluation is marked when the root flag, any prerequisite flag read during evaluation, or any segment loaded for rule matching carries the override marker; unresolved flags/segments do not count, and segment reads count even when the segment does not match (including negated clauses and nested segment rules). Prerequisite evaluation uses scoped marking that **merges upward** so sibling prereq events stay unmarked while affected subtrees and the top-level result reflect overrides; error outcomes are still marked if an override definition was read. > > Extensive tests cover model serialization, marking rules, and parity between sync and async evaluators. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1debdc2. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
<!-- TERMAG_PR_BODY_BEGIN --> ## Summary This is the third step of the flag overrides port described by the OVERRIDE specification. It is based on the model marker branch because the phases are stacked; retarget to `feat/overrides` once that branch merges. **The override layer** (`ldclient/impl/overrides`). `OverrideLayer` is the override store: it holds marked shallow copies of the flag and segment definitions a source supplies, keyed by key, and is replaced wholesale on each update, so the layer's contents are exactly one snapshot at any instant and an empty snapshot clears it. Dictionaries are decoded with the model constructors, and an invalid definition raises rather than being defaulted. A definition whose key differs from its key in the snapshot raises too, and the layer is left unchanged: evaluation reads an entry by its snapshot key, while the all-flags state and events use the definition's key. `OverrideStoreView` is the overlay at the store read boundary: a read of a flag or segment returns the override entry when one exists and the LaunchDarkly entry otherwise, whatever the state of the base store, and an enumeration is the union of both with the override entry winning, including over a deleted item. A failure of the base enumeration propagates, as it does without the overlay. When that read fails, `all_flags_state` logs the failure as before and, if the override layer holds flags, returns a state of those flags marked invalid rather than an empty one: overrides only add to the state, and a client-side SDK bootstrapped from it uses its flags whatever `$valid` says, so the overridden values still reach it. With no override flags the state is the same invalid, empty state as without a source. The sink stops accepting snapshots when the client is closed, and when the source fails to start it also clears what the source supplied. `OverrideSinkImpl` applies each snapshot as one serialized operation and notifies flag change listeners of every flag whose merged-view evaluation may have changed: entries added, removed, or changed, plus every flag that depends on them through prerequisites or segment references. An identical snapshot notifies nothing. **LaunchDarkly changes reach override dependents.** The layer keeps a dependency graph of its entries, rebuilt on each replacement, and the store (sync and async) expands every changed LaunchDarkly item through both its own graph and the layer's, to a fixpoint. A change to a prerequisite flag or a segment that an override entry references therefore notifies that entry's flag, the same way dependent flags are notified for LaunchDarkly data, and a chain may cross between LaunchDarkly data and overrides any number of times. Without an override source the store's behavior is unchanged. **Configuration and wiring.** `OverrideSource` and `OverrideSink` are public protocols in `ldclient.interfaces`, so a custom source can be supplied. `OverrideSourceBuilder` and `DataSystemConfig.override_source` (also on `AsyncDataSystemConfig`) plus `ConfigBuilder.overrides(...)` make the source an option of the FDv2 data system. The source is built with the data system, so invalid configuration raises from the client constructor like any other invalid component configuration. It starts before the data system's own threads and its initial load completes before the constructor returns; it is closed when the client is closed, after the sink stops accepting snapshots, and an exception from its `close` is logged. An exception from its `start` does not fail the constructor or `start`: the data system logs it, closes the source, clears any snapshot the source supplied before failing, and the client runs without overrides. The SDK's own file-based source does not raise from `start`; a custom source could. The source is not built or started when the client is offline, and the SDK logs a warning that it is not used. The data system's `store` becomes the overlay when a source is configured, so evaluation, prerequisite and segment resolution, and the all-flags read all see override precedence. The override source has no effect on initialization status, data availability, or data source status. The legacy FDv1 data system reports no override source. When the data system fails to start, the client constructor stops each component in turn and logs any failure to stop, so the event processor still stops and the caller receives the start failure. **The client facade.** Both `LDClient` and `AsyncLDClient` consult the override layer before the not-initialized short-circuit: an overridden flag is served before the client has LaunchDarkly data, and a flag the layer does not hold takes exactly the path it took before this feature, so a configured but empty source changes nothing with the same warning and event as before. The all-flags state reads through the overlay. While the client has no LaunchDarkly data it contains only the flags the layer holds, read from the layer rather than the overlay so that LaunchDarkly data in a store without an init marker stays out (with a once-per-client warning), and it is invalid when the layer holds no flags. Without a source this state is empty, because the SDK reports no flags before the store is initialized, so the override flags only add to what it reports; overrides never remove a flag from the state. An evaluation that fails with an unexpected exception reports the `EXCEPTION` reason with `overrideAffected` when the flag came from the override layer, in a single evaluation and in the all-flags state (OVERRIDE 1.5.4); the marker is read through `is_override_definition`, so the error path cannot fail on an unexpected store item. On the async client, override change notifications are marshalled onto the event loop, like every other change notification. **The async client with a persistent store.** The async client works out which flags an override change affects from the in-memory store, the only store it can read synchronously (`async_fdv2.py`). While a persistent store is the active store, a change to an override notifies the overridden flags, but not LaunchDarkly flags that depend on them through a prerequisite or a segment. The documentation of `AsyncDataSystemConfig.override_source` says so. **Left to later PRs in the stack.** - `all_flags_state` still keeps `trackEvents`, `trackReason` and `debugEventsUntilDate` for override-affected flags (OVERRIDE 1.4.1.1). The events step, #528, clears them. - An override entry marked as deleted is rejected by #574, which is stacked on top of the stack (launchdarkly/sdk-specs#288). **Tests.** Unit tests for the layer, overlay (sync and async), and sink; client tests for the gate, source lifecycle, construction errors, offline, precedence, prerequisite and segment overrides, all-flags state, and flag change and flag value change listeners, for both clients; and a runner for the OVERRIDE specification test vectors (`ldclient/testing/testdata/override-vectors/vectors.json`) through the full client stack. The vector runner's per-evaluation summary marker assertion is added with the events step. [SDK-3250](https://launchdarkly.atlassian.net/browse/SDK-3250) [SDK-3250]: https://launchdarkly.atlassian.net/browse/SDK-3250?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Adds **experimental flag/segment overrides** that can take precedence over LaunchDarkly data at evaluation time, wired through FDv2 (sync and async). > > **Configuration:** Optional `override_source` on the data system (`ConfigBuilder.overrides(...)`, `OverrideSourceBuilder`, public `OverrideSource` / `OverrideSink` protocols). Offline clients skip building/starting the source. > > **Runtime:** An `OverrideLayer` holds snapshot replacements; `OverrideStoreView` overlays the feature store so per-key reads prefer overrides; `OverrideSinkImpl` applies updates and fans out **flag change** notifications via merged-view dependency diffing. The store can register the layer as **external dependencies** so LaunchDarkly prerequisite/segment updates also notify affected override flags. > > **Clients:** `LDClient` / `AsyncLDClient` read the override layer **before** the not-initialized default path and adjust **`all_flags_state`** to return override-only bootstrapping when LD data is unavailable (with once-per-client warnings). Evaluation errors set **`overrideAffected`** when the definition came from overrides. Sync client **rolls back components** if data system start fails. > > **Tests:** Layer/overlay/sink units, broad sync+async client scenarios, store cross-graph notification, and OVERRIDE **spec vectors** (`override-vectors/vectors.json`). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 0489514. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- TERMAG_PR_BODY_END --> <!-- termag: box=i-0575d1c54853058d7 project=Overrides --> <sub>Opened from termag</sub>
## Summary This is the fourth step of the flag overrides port described by the OVERRIDE specification. It is based on the override store branch because the phases are stacked; retarget to `feat/overrides` once that branch merges. `EventInputEvaluation` gains `override_affected`, set by the client for the top-level evaluation and by the evaluator for each prerequisite record from that record's own marking. The event processor keys on this scalar alone and never reads the evaluation reason: an override-affected evaluation produces no individual feature event and no debug event, even when the flag requests them, while its index event and its summary counting are unchanged. An unaffected prerequisite record inside a marked evaluation is still recorded as usual. The summary counter key becomes `(variation, version, override_affected)`, so override-affected and ordinary evaluations of the same flag, variation, and version accumulate into separate counters instead of collapsing together. A counter that aggregates marked evaluations carries `overrideAffected: true` in the summary event; the property is present only when true, like the `unknown` marker. Both the sync and async event processors share this dispatch and output code. In the all-flags state, an override-affected flag keeps its value, version, and marked reason but is presented with `trackEvents` and `trackReason` off and no `debugEventsUntilDate`, so a consumer bootstrapped from the state sends no individual events for it. When a migration evaluation of an overridden flag replaces the reason with a wrong-type error, the new reason keeps the marking. The async client mirrors these changes. The OVERRIDE specification vector runner now also asserts the `summaryOverrideAffected` field of each vector against the marking the client hands to the event processor. Tests cover the summarizer key split, the marker in the summary output, suppression of feature and debug events through both real event processors (including a marked prerequisite record and a mixed set of evaluations of one flag), the client-side marking of top-level and prerequisite records through a prerequisite tree, the all-flags tracking fields, the wrong-type reason, and an end-to-end payload check that an override-affected flag appears only in the summary. [SDK-3250](https://launchdarkly.atlassian.net/browse/SDK-3250) [SDK-3250]: https://launchdarkly.atlassian.net/browse/SDK-3250?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ An evaluation that fails with an unexpected exception keeps the marking when the flag came from the override layer, so it produces no individual event and the all-flags state presents it with tracking off. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Adds an **`override_affected`** marker on evaluation events so flag-override evaluations are handled differently in analytics. > > **Event pipeline:** Override-affected evaluations still update **summary counters** (with `overrideAffected: true` when marked) and index events, but **no individual feature or debug events** are emitted even when tracking is on. Summary counter keys are now `(variation, version, override_affected)` so marked and unmarked evaluations don't share the same bucket. > > **Clients (sync + async):** Evaluation paths pass `override_affected` into `EventFactory`; **WRONG_TYPE** and **exception** paths preserve the override marking when applicable. **`all_flags_state`** clears `trackEvents`, `trackReason`, and `debugEventsUntilDate` for override-affected flags so bootstrapped consumers don't fire individual events. Prerequisite evaluation records carry their own override marking. > > **Async FDv2:** Late override reloads after client close no longer error when the event loop is closed—notifications are dropped with a debug log. > > Specification vector runners and new tests cover summarizer/output behavior, event suppression, prerequisite propagation, and end-to-end payloads. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1588f31. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
<!-- TERMAG_PR_BODY_BEGIN -->
## Summary
This is the fifth step of the flag overrides port described by the
OVERRIDE specification. It is based on the events branch because the
phases are stacked; retarget to `feat/overrides` once that branch
merges.
`ldclient.integrations.overrides.FileOverrideSourceBuilder` is the
public entry point. It is passed to
`datasystem.ConfigBuilder.overrides(...)` and builds an override source
that reads one or more local files in the file data source document
format (optional `flags`, `flagValues`, and `segments` members, JSON or
YAML) and supplies each successful load to the SDK's override sink as a
full snapshot. `flagValues` entries expand into full flag definitions,
so the layer holds only full entities; as the specification describes
for a value-only override, the expansion is a flag that is on and serves
its single variation as the fallthrough, so the evaluation reason kind
is `FALLTHROUGH`. The existing file data sources keep their current
behavior, including their own `flagValues` expansion.
Behavior:
- The file name chooses each file's format: a name that ends in `.yaml`
or `.yml` is YAML, which needs the `pyyaml` package, and any other file
is JSON. Files must be UTF-8. A YAML file configured without `pyyaml` is
a construction error, since it could never load. An empty JSON file is
not a valid document, nor is JSON that is not an object, such as `null`;
an empty YAML file has no entries.
- Files are combined in the configured order. `duplicate_keys_handling`
is `fail` by default, which rejects the reload and keeps the previously
loaded overrides, or `ignore`, which keeps the first configured file's
entry.
- A configured file that does not exist contributes no overrides and is
not an error: it can be created later, deleting a file removes its
overrides, and deleting the last file clears the layer. Writing `{}`
also removes a file's overrides.
- Change detection is in place before the initial load, so a change made
between the two is not missed.
- A file that exists but cannot be read or parsed fails that whole
reload. The last good overrides stay in effect, the failure is logged,
and the load is retried after a bounded delay and on the next detected
change, so a file observed mid-write recovers on its own.
- Files can be rewritten in place. While the source finds a file empty
during a rewrite, it keeps the overrides in effect until the new content
is written. The hold that does this is part of the reloader in #525: a
file whose content is in effect and that a load finds with no entries is
held back until it has stayed so, unwritten, for the retry delay (one
second), each file for at most five retry delays. After that, an emptied
YAML file removes its overrides and an empty JSON file fails to load and
keeps them. A partly written YAML file can still parse, so the docs
recommend replacing a YAML file in one operation.
- `change_detection` is one of two alternatives. `polling` (the default)
compares modification time and size on an interval, one second by
default with a one second minimum; a lower interval is raised to the
minimum with a warning, and a value that is not a finite number, or
longer than the runtime can wait, is a construction error. `watching`
uses the `watchdog` package and is a construction error when the package
is not installed. Notifications are debounced so a burst from one edit
produces one reload.
- The initial load runs synchronously inside `start`, which the data
system calls during client construction, so an override present at
startup takes effect from the first evaluation.
- Every applied change is logged at Info level, for example `Flag
overrides in effect: 2 flags, 1 segment (/etc/ld/a.json: 2 flags, 1
segment; /etc/ld/b.json: absent)`, or `Flag overrides: none in effect
(...)`.
- A builder with no paths is a construction error. Unknown
`change_detection` or `duplicate_keys_handling` values raise
`ValueError` when set. This is stricter than the Go builder, which
treats an unrecognized duplicate keys handling as `fail`. A path listed
more than once is logged at warning level and the repeats are ignored;
the first occurrence keeps its place in the order. Paths are compared
the way the platform compares file names (`os.path.normcase`): without
regard to case on Windows, exactly elsewhere.
The module is added to the API reference (`docs/api-integrations.rst`)
and every public docstring carries the experimental note. The public
docs say what an application needs (the format rule, UTF-8, in-place
rewrites, how to remove overrides, replacing YAML in one operation); the
hold's mechanics are described in the internal docstrings and here.
Tests cover the builder validation and defaults (including a YAML file
without `pyyaml`), synchronous initial load, YAML, multi-file order and
duplicate handling, absent files appearing and disappearing, the Info
log lines, both change detection modes, last-good retention across a
malformed edit in both modes, the automatic retry with no change signal,
close semantics, and an end-to-end run through `LDClient` where a file
is added, changed, and cleared with `{}` while the client never receives
LaunchDarkly data. In-place rewrite tests, for JSON and YAML in watching
and polling modes, rewrite the file slowly (truncate, pause, write) and
check that the source never supplies a snapshot without the override and
that a client evaluating the flag throughout serves the old value and
then the new one, never the default; they also cover successive rewrites
at the source's own timing, an emptied JSON file keeping the overrides,
and an emptied YAML file removing them after the retry delay.
The in-place rewrite handling was reviewed in eight rounds as #560 (now
closed, its changes moved here and into #525). The final mutation check
ran 59 mutants against `test_filedata.py` and
`test_file_override_source.py`: 54 fail a test, and the 5 survivors are
equivalent in this code. Both test files ran 10 times in a row with no
failure, and the harness's `flag overrides` suite passes against the
sync and async test services at the top of the stack.
[SDK-3250](https://launchdarkly.atlassian.net/browse/SDK-3250)
[SDK-3250]:
https://launchdarkly.atlassian.net/browse/SDK-3250?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
<!-- TERMAG_PR_BODY_END -->
<!-- termag: box=i-0575d1c54853058d7 project=python-server-sdk -->
<sub>Opened from termag</sub>
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Overview**
> Adds an experimental **file-based flag override source** so operators
can force flag/segment values from local JSON or YAML files (same format
as the file data source) via `FileOverrideSourceBuilder`, wired through
`datasystem.ConfigBuilder.overrides(...)`.
>
> The source loads an initial snapshot synchronously at client startup,
merges multiple files with configurable duplicate-key handling, and
reloads on change using **polling** (default) or optional **watchdog**
watching. It tolerates missing files, parse failures (keeps last good
overrides with retry), and in-place rewrites without briefly clearing
overrides. Applied changes are logged at Info level. API docs for
`ldclient.integrations.overrides` are added; behavior is covered by
extensive integration tests including end-to-end `LDClient` scenarios.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
9a582aa. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## Summary This is the last step of the flag overrides port described by the OVERRIDE specification. It is based on the file source branch because the phases are stacked; retarget to `feat/overrides` once that branch merges. Both contract test services (the Flask service for `LDClient` and the aiohttp service for `AsyncLDClient`) advertise the `flag-overrides` capability and accept the `overrides` object in the SDK configuration as the test harness service specification describes: `filePaths` (required, in precedence order) and the optional `duplicateKeysHandling`, `changeDetection`, and `pollIntervalMs` properties. The values configure a `FileOverrideSourceBuilder` on the FDv2 data system through `ConfigBuilder.overrides(...)` or `AsyncDataSystemConfig.override_source`. An `overrides` object without a `dataSystem` configuration is rejected with an error, because overrides are an option of the FDv2 data system only. Verification with the prebuilt harness from sdk-test-harness branch `rlamb/sdk-2657/flag-overrides-tests` (v3.2.0-alpha.10): - Sync service: `-run 'flag overrides'` 38 ran, all passed; full suite 4809 total, 15 skipped, 4794 ran, all passed. - Async service: `-run 'flag overrides'` 38 ran, all passed; full suite 4807 total, 16 skipped, 4791 ran, all passed. The 15 tests skipped by the overrides runs are the other top-level suites excluded by the filter. The tests skipped in the full runs are those for capabilities the services do not advertise (`strongly-typed`, `tls:verify-peer`, `service-endpoints`, `user-type`, `track-hooks`, `wrapper`, and on the async service `hook-environment-id`), the long-running tests, and the persistent data store tests, all of which were skipped before this change too. [SDK-3250](https://launchdarkly.atlassian.net/browse/SDK-3250) [SDK-3250]: https://launchdarkly.atlassian.net/browse/SDK-3250?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Both contract-test services (sync Flask and async aiohttp) now **advertise** the `flag-overrides` capability and map harness `overrides` config into the FDv2 data system. > > When `dataSystem` is present, an optional `overrides` block builds a `FileOverrideSourceBuilder` from `filePaths` and optional `duplicateKeysHandling`, `changeDetection`, and `pollIntervalMs`—wired via `datasystem.overrides(...)` on the sync client and `AsyncDataSystemConfig.override_source` on the async client. **`overrides` without `dataSystem` raises `ValueError`**, since overrides are FDv2-only. > > No production SDK behavior changes; this is harness wiring so sdk-test-harness flag-overrides suites can run against Python. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit dac6abe. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
<!-- TERMAG_PR_BODY_BEGIN --> **Requirements** - [x] I have added test coverage for new or changed functionality - [x] I have followed the repository's [pull request submission guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests) - [ ] I have validated my changes against all supported platform versions - [x] Sync/async parity: matching changes made to `async_*` siblings (or N/A). N/A: `LDClient` and `AsyncLDClient` use the same `FileOverrideSourceBuilder`. **Related issues** - Bugbot finding on #529: #529 (comment) **Describe the solution you've provided** `FileOverrideSourceBuilder` accepted one path as a `str`, or a list of paths. A single `pathlib.Path`, or any other `os.PathLike`, was passed to `list()`. A path object is not iterable, so the constructor raised `TypeError: 'PosixPath' object is not iterable`. The Bugbot finding describes a different failure, a path split into its components that fails every reload, but that cannot happen. The constructor raises before a client exists. The constructor now treats a `str` or an `os.PathLike` as a single path, and converts every path with `os.fspath`. A path that does not name a string path raises `TypeError` from the constructor, with a message that names the value. That includes `bytes` (given alone, in a list, or named by a path object), an `int`, and `None`. `bytes` and `bytearray` count as single values, so a bare `bytes` path is named whole instead of being read as a sequence of integers. The type hint is `Sequence` rather than `List`, so that type checkers accept a list of path objects but not a set, whose order is not fixed. The docstring says that paths may be path objects. Tests cover: - a single path object; - a list of path objects, which already worked; - a path object whose string form differs from its path, as with `os.DirEntry`; - a source that loads a file given as a path object; - the rejection of each of the values above. Each of six broken variants of the constructor fails at least one of these tests, including the original code and a variant that converts with `str()` instead of `os.fspath`. **Describe alternatives you've considered** Leaving the builder to accept only strings, as its type hint said. `pathlib` is the usual way to build paths, a list of path objects already worked, and the error did not say what was wrong. **Additional context** This PR is stacked on #530, the top of the overrides stack, so that the approved PRs below it do not change. Retarget it as the PRs below it merge. The review of this change found three existing gaps that this PR leaves alone: - An empty path resolves to the working directory, and its reads keep failing. - In polling mode, a path that contains a NUL character makes the source fail to start, so the client runs without overrides. In watching mode, the reload fails and keeps retrying. - The file data sources treat a tuple of paths differently from a list. Testing: `mypy ldclient`, `isort --check` and `pycodestyle` pass locally. The test suite passes (2,111 tests). It was run without the Redis, DynamoDB and Consul store integration tests, which need servers that were not running. CI runs only for PRs based on `main` or `feat/**`, so it will not run on this PR until it is retargeted. <!-- TERMAG_PR_BODY_END --> <!-- termag: box=i-0575d1c54853058d7 project=Overrides --> <sub>Opened from termag</sub> <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Fixes a constructor bug where a **single** `pathlib.Path` (or other `os.PathLike`) was passed to `list()`, causing `TypeError: 'PosixPath' object is not iterable`. > > `FileOverrideSourceBuilder` now wraps a lone `str`, `bytes`/`bytearray`, or `PathLike` as one path, normalizes every entry with `os.fspath()` (so objects like `os.DirEntry` use their filesystem path, not `str()`), and stores string paths. Invalid inputs (`bytes` paths, ints, `None`, path objects that do not resolve to a `str`) fail fast with an explicit `TypeError`. Type hints use `Sequence` plus `PathLike` so callers can pass path objects while unordered collections like `set` are discouraged. > > New tests cover single/list path objects, `__fspath__` behavior, invalid types, and loading overrides from a path object. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 154b5b8. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
<!-- TERMAG_PR_BODY_BEGIN --> **Requirements** - [x] I have added test coverage for new or changed functionality - [x] I have followed the repository's [pull request submission guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests) - [ ] I have validated my changes against all supported platform versions (run on Python 3.13 only; CI does not run for this base) - [x] Sync/async parity: matching changes made to `async_*` siblings (or N/A). The fix is in the override layer and sink, which `LDClient` and `AsyncLDClient` share. The tests cover both clients. **Related issues** - Implements the "reject deleted entries" rule of launchdarkly/sdk-specs#288 (OVERRIDE Requirements 1.1.1.1, 1.2.8.1 and 1.3.6.2). - Fixes the deleted-entry defect found in the review of #527. - Stacked on #572 (`rlamb/overrides-python-file-source-paths`) as a new PR, so the PRs below it do not change. **Describe the solution you've provided** Before this change, a flag or segment with `"deleted": true` went into the override layer, from a custom source or from an override file. The file source counted the entry as an override in effect. The overlay served it, and `variation()` and `variation_detail()` raised `AttributeError: _track_events` into the application, in both `LDClient` and `AsyncLDClient`. A flag that depended on it evaluated with reason `EXCEPTION` and without `overrideAffected`. After this change: - **Sink and layer (1.1.1.1, 1.2.8.1).** When a snapshot contains a flag or segment marked as deleted, as a dictionary or a model object, the sink rejects the whole snapshot. The layer is unchanged and no change notifications are sent. The rejection is logged at Error level and names the kind and the key, for example `Rejected an override snapshot; the overrides in effect stay: flag "f" is marked as deleted; an override is removed by leaving it out of the snapshot`. `set_overrides` raises `ValueError`, as it already does for an invalid definition, so the data system handles both the same way. If a source lets the error propagate out of `start`, the existing start-failure handling applies: the error is logged, the source is closed, and the client runs without overrides. A rejected later update keeps the overrides in effect. The error goes back to the source's own caller. - **File override source (1.3.6.2).** The source checks for deleted entries when it parses each file, before it calls the sink. The reload fails the same way as for a file that cannot be parsed. The previous overrides stay, and the failure is logged like other load failures, naming the file and the key: `Unable to load flag data: flag "f" in /path/overrides.json is marked as deleted; remove the entry to remove its override`. The existing retry applies. - An entry with `"deleted": false` is an ordinary definition and still loads. - **Docs.** `FileOverrideSourceBuilder` and `OverrideSink.set_overrides` each get one user-level sentence. **The ordinary file data sources are unchanged on purpose.** `Files.new_data_source` and `Files.new_data_source_v2` do not use the override source's parsing code, and they still accept deleted entries. Rejecting deleted entries there would be a breaking change, so it is left for a later major version. New tests in `test_file_data_sources_pinned_behavior.py` pin the current behavior: the FDv1 source stores such entries as deleted, and the FDv2 source passes them on unchanged. **Tests** - `testing/impl/test_overrides.py`: the layer and the sink reject a deleted flag and a deleted segment, each as a dictionary and as a model, with and without listeners. In every case the layer is unchanged, nothing is notified, exactly one Error is logged, and the next snapshot is accepted. A definition with `deleted: false` is accepted. - `testing/test_ldclient_overrides.py` and `testing/test_async_client_overrides.py` use a custom source. If it supplies a deleted flag at start, the client serves LaunchDarkly data for the flag and its dependent. If it supplies one in a later update, `ValueError` is raised to the source, the override and the dependent's `overrideAffected` reason stay, and nothing is notified. In neither case do `variation()` or `variation_detail()` raise. - `testing/integrations/test_file_override_source.py`: - An initial load with a deleted flag or segment fails, in JSON and in YAML, and applies none of the other file's overrides. - A reload with a deleted entry keeps the previous overrides and logs the file and the key. The retry then picks up the fix with no change signal. - `deleted: false` loads. - A client with such a file serves no override. - `testing/integrations/test_filedata.py`: `load_files` and `read_file` raise `DeletedEntryError`, a `FileReadError`, and its message names the file and the key. `deleted: false` loads. - `testing/integrations/test_file_data_sources_pinned_behavior.py`: pins that the ordinary data sources are unchanged. I ran every rejection test against the base commit without the fix, and each one fails there. The `deleted: false` tests and the pins pass both with and without the fix, as expected. How I ran them, with the clone's venv and this worktree on `PYTHONPATH`: ``` python -m pytest -q -p no:cacheprovider ldclient --ignore-glob='*dynamodb*' --ignore-glob='*redis*' --ignore-glob='*consul*' # 2142 passed mypy ldclient # no issues isort --check --atomic ldclient contract-tests # clean pycodestyle ldclient contract-tests # clean ``` **Describe alternatives you've considered** - Dropping only the deleted entry and applying the rest. The spec requires rejecting the whole snapshot, so that the layer always holds exactly one snapshot that a source supplied. - Serving a deleted entry as an absent flag. That would hide LaunchDarkly data for the key, and overrides only add or replace definitions. **Additional context** Python CI runs only for PRs based on `main` or `feat/**`, so it does not run on this PR. <!-- TERMAG_PR_BODY_END --> <!-- termag: box=i-0575d1c54853058d7 project=python-server-sdk --> <sub>Opened from termag</sub> <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Override snapshots and file-based override loads now **reject** any flag or segment with `"deleted": true`, instead of accepting them into the override layer (which could break evaluation). > > **File parsing (`filedata`)** adds `DeletedEntryError` and `_reject_deleted_entries`, run after parse in `read_file` / `_parse_file`, so a reload with a deleted entry fails like a parse error and prior overrides stay in effect. > > **Override application** adds `DeletedItemError` in `OverrideLayer.set_all`; `OverrideSinkImpl` logs and re-raises so nothing is applied and listeners are not notified. Public docs on `FileOverrideSourceBuilder` and `OverrideSink.set_overrides` state that removal is done by **omitting** the entry, not by marking it deleted. > > **Legacy file data sources (FDv1/FDv2)** are unchanged and still accept deleted entries; new pinned tests document that behavior. > > Tests cover layer, sink, sync/async clients, file override source (JSON/YAML), and `load_files`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit c6dcabc. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
…ees the malformed write (#576) <!-- TERMAG_PR_BODY_BEGIN --> **Requirements** - [x] I have added test coverage for new or changed functionality - [x] I have followed the repository's [pull request submission guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests) - [ ] I have validated my changes against all supported platform versions - [x] Sync/async parity: matching changes made to `async_*` siblings (or N/A). N/A: a test-only change to the file override source tests. **Related issues** `test_source_retries_a_failed_load_without_a_change_signal` failed on `windows (3.10)` in the CI of #572, with `AssertionError: the load failure was not logged`. The test passed on every other Windows and Linux version. **Describe the solution you've provided** The test writes valid content, then malformed content, then fixed content. The fixed content keeps the malformed content's size and modification time, so that only the automatic retry can recover. The initial content had that same size too. The poller compares a file's modification time and size. Where file times are coarse, as on Windows, the initial and malformed writes can share a modification time, so the poller saw no change. Nothing was then logged, and the test timed out. The initial content now has a different size from the malformed content, so the poller always sees the malformed write. The part of the test that depends on an unchanged size and time is unchanged. To confirm the cause, I patched `_observe` to round modification times down to whole seconds and ran the test. It failed after its 10-second timeout without this change, and it passes with it. The whole test file passes normally (72 tests), and `isort` and `pycodestyle` are clean. **Describe alternatives you've considered** Setting the file's modification time after the malformed write. Changing the size states the intent in the content itself, and needs no extra system call. **Additional context** This merges cleanly with #574, which adds tests to the same file. <!-- TERMAG_PR_BODY_END --> <!-- termag: box=i-0575d1c54853058d7 project=python-server-sdk --> <sub>Opened from termag</sub> <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Stabilizes **`test_source_retries_a_failed_load_without_a_change_signal`** on Windows by changing the **initial** override JSON so its **file size differs** from the later malformed payload. > > The test still relies on malformed and fixed content having the **same length and mtime** so recovery must come from **automatic retry**, not polling. On platforms with coarse mtimes, the old initial content could match the malformed write on size and time, so the poller missed the bad edit and the expected load-failure log never appeared. Comments were added to document that intent. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 0d9ba1c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
…to a touch (#577) <!-- TERMAG_PR_BODY_BEGIN --> **Requirements** - [x] I have added test coverage for new or changed functionality - [x] I have followed the repository's [pull request submission guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests) - [ ] I have validated my changes against all supported platform versions - [x] Sync/async parity: matching changes made to `async_*` siblings (or N/A). N/A: the file watcher serves both clients. **Related issues** This resolves the SDK findings of the Windows stability assessment for the overrides work: [plans/python-overrides-windows-test-stability.md](https://github.com/launchdarkly/sdk-scratchpad/blob/main/plans/python-overrides-windows-test-stability.md) in sdk-scratchpad. - Finding 1, the flaky retry test, is fixed by #576. - Findings 2, 3 and 4 are fixed here. **Describe the solution you've provided** All three changes are in `Watcher`, in `ldclient/impl/integrations/files/filedata.py`, which the file-based override source uses in watching mode. 1. **watchdog 7 delays Windows events by a second, and starves them in a busy directory** (Finding 2). - watchdog 7.0.0 (2026-10-31) changed its Windows reader. It now waits for the observer timeout after the first notification before delivering anything, and it delivers nothing while notifications keep arriving within that timeout. - With the default one-second timeout, a change to an override file was reported about a second late on Windows. It was not reported at all while another file in the same directory, such as a log, changed more often than once a second. - The assessment measured, on Windows Server 2025 with watchdog 7.1.0: a median latency of 1003 ms, and three of three changes not reported within 8 s next to a file appended every 200 ms. With a 100 ms timeout: 102 to 110 ms, and 3 of 3 reported. - The watcher now builds the observer with `timeout=0.1`. On Linux, the timeout only sets how often the idle inotify loop wakes. - Python 3.11 and later install watchdog 7 under `watchdog>=3.0.0`, so this affects any Windows deployment that selects `ChangeDetection.WATCHING`. 2. **A touch no longer reloads with watchdog 7** (Finding 3). - watchdog 7 reports a change to a file's metadata, such as a new modification time from `os.utime`, as an `attrib` event. Earlier versions reported it as `modified`. - The watcher ignored `attrib`, so touching a file to force a reload stopped working in watching mode, while it still works in polling mode. - `attrib` now counts as a change. A plain read does not produce it, so the reloads stay quiet. The reloader's content digest still skips applying an unchanged file. 3. **watchdog 6 on Windows can miss a change right after a watch starts** (Finding 4). - watchdog 6.0.0, the last version that supports Python 3.10, returns from starting a watch after a fixed 10 ms. Its reader thread may not yet have asked for the first notification, and anything before that request is lost. - The assessment found that a deletion made immediately after the watcher was created was reported 0 of 10 times, and 10 of 10 times after one thread yield. - The watcher now yields once after each watch it sets up. Tests in `test_filedata.py`: - a change is reported within 0.5 s; - a change is reported within 2 s while another file in the directory is appended to five times a second; - a change of the modification time alone is reported. **Describe alternatives you've considered** - Pinning `watchdog<7` in the `test-filesource` extra. Applications bring their own watchdog, so a pin would only hide the behavior from CI. - Running a slow safety poll under watching mode, so that a lost or starved notification is bounded. That is a design decision for the spec, not part of this fix. - Documenting that watching mode ignores a touch, instead of reacting to `attrib`. That would leave watching and polling behaving differently for the same file. **Additional context** Testing: - On Linux with watchdog 7.1.0, the touch test fails without this change and passes with it. - The latency and busy-directory tests pass on Linux either way, because inotify delivers each event at once. Windows CI runs them against watchdog 7, which they target. - Every watcher test passes with watchdog 6.0.0 and with watchdog 7.1.0. That includes the existing test that a read alone does not signal. - The file data and override source modules pass on watchdog 7.1.0 (263 tests). - The full suite passes (2,120 tests), without the Redis, DynamoDB and Consul store tests, which need servers that were not running. - `mypy`, `isort` and `pycodestyle` are clean. This PR merges cleanly with #574 and #576. Not in this PR, from Finding 6 of the assessment. None of these come from the overrides work: - a latency assertion in `test_async_migrator.py` that is too tight for Windows clocks; - a streaming retry test that is flaky on `main`; - the big segments status-polling contract test, which turns Linux CI on `feat/overrides` red in 4 of 5 runs and needs its own investigation before the merge into `main`. The assessment also recommends reporting the watchdog 7 behavior upstream; that is not done yet. <!-- TERMAG_PR_BODY_END --> <!-- termag: box=i-0575d1c54853058d7 project=python-server-sdk --> <sub>Opened from termag</sub> <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Updates the file **`Watcher`** in watching mode so override/data reloads behave reliably with **watchdog 6–7**, especially on Windows. > > The observer now uses a **100 ms** delivery timeout instead of watchdog’s 1 s default, so changes are reported quickly and are not indefinitely delayed when another file in the same directory keeps notifying. **`attrib`** is treated as a content-relevant event so a **mtime-only touch** triggers a reload again under watchdog 7. After each directory watch is scheduled, a **thread yield** (`time.sleep(0)`) avoids missing an immediate post-start change on watchdog 6 for Windows. > > New watcher tests cover sub-second latency, detection alongside a busy log file, and mtime-only updates. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 546a7f5. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
Conflicts in the capability lists of the contract test services keep both sides: flag-overrides from feat/overrides, and the retry conformance, service endpoints, wrapper, proxy and TLS capabilities from main. The async override test that captured evaluation details used AsyncHook, which #548 removed from main; it now uses Hook, which the async client accepts.
keelerm84
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BEGIN_COMMIT_OVERRIDE
feat: Add experimental support for flag overrides, read from local JSON or YAML files or supplied by a custom override source
feat: Mark evaluations that a flag override affected with
overrideAffectedin the evaluation reason, and count them under separate summary countersfix: Stop the components that already started when the client fails to start
END_COMMIT_OVERRIDE
Every change in this PR was already reviewed and merged through its own PR against
feat/overrides: #525, #526, #527, #528, #529, #530, #572, #574, #576, #577. This PR merges the branch intomain.Flag overrides let an operator force flags and segments to known definitions on a running client, whether or not the client can reach LaunchDarkly, and remove the override when the incident is over. They are an option of the FDv2 data system, on both
LDClientandAsyncLDClient, with a file-based source inldclient.integrations.overrides.FileOverrideSourceBuilder, and are experimental.The one change outside the feature: #527 stops the components that had already started when the data system fails to start, so a failed client construction no longer leaves threads behind.
mainwas merged intofeat/overridesin 46e5f0e, so the branch has everymaincommit. That merge resolved two conflicts and made one follow-up change:flag-overridesfrom this branch, and the retry conformance, service endpoints, wrapper, proxy and TLS capabilities frommain.AsyncHook, which feat: Accept synchronous hooks on the async client #548 removed frommain. It now usesHook, which the async client accepts.Before the push:
mypy,isortandpycodestylewere clean;flag overridessuite passed against both the sync and the async test service (88 tests each).Opened from termag
Note
Overview
Adds experimental flag overrides so operators can force known flag/segment definitions at runtime (via local JSON/YAML files through
FileOverrideSourceBuilder, or a customOverrideSource), integrated into the FDv2 data system for sync and async clients throughdatasystem_config.override_source/ConfigBuilder.overrides().The data system now maintains an override layer merged over the normal store, serves override-only flags when LaunchDarkly data is not ready, and fans out dependency-aware flag-change notifications when LD data affects overridden prerequisites/segments. Evaluations propagate an
overrideAffectedreason marker; analytics treat those evaluations differently (summary counters only,all_flags_statedisables per-flag tracking for them).Contract test services advertise
flag-overridesand build override sources from harness config; docs addldclient.integrations.overrides.Also fixes client startup: if the data system fails to start, already-started components are stopped so failed construction does not leak threads.
Reviewed by Cursor Bugbot for commit 46e5f0e. Bugbot is set up for automated code reviews on this repo. Configure here.